The generation of a panorama background from a set of still images or a sequence of video frames (collectively “images”) is known. Stationary content from the images of the set relating to the same physical scene at a plurality of different time instances, viewpoints, fields of view, resolutions, and the like (that is, “background”) is melded to form a single, wider angle background image (or “panorama background”). Panorama background generation is very useful in many applications, ranging from object tracking in surveillance systems, to noise reduction for weather pattern analysis and astronomical surveying.
Panorama background generation is a specific form of panorama generation, wherein foreground objects moving across the image background are treated as noise and removed. In order to generate a panorama background, the images of the set are geometrically and/or colorimetrically registered and aligned, and stationary portions of the images in the set are then identified and merged or stitched together to form a view of the scene as a single coherent panorama background. During registration, each image is analyzed to determine if it can be matched with one or more previously analyzed images. A transformation model between the image and a reference image is determined and then the image is warped to the other images to remove or minimize the offset.
There are a number of issues that must be taken into account during the generation of a panorama background in order to ensure a good resultant image. Background motion requires tolerance to image misalignments, which is not a major concern for most object tracking and surveillance algorithms making these algorithms unsuitable for panorama background generation. In panorama background generation, background movements can be relatively large and thus, tolerance to image misalignment is required. Therefore, a more flexible approach is typically required to provide greater tolerance to such image misalignment.
In general, during panorama background generation, it is often very hard to obtain clean pictures of landscapes and landmarks due to the noise introduced by moving foreground objects. For example, taking clean pictures of a park for panorama background generation can often be difficult due to the large number of people moving in front of the camera. Current techniques of panorama background generation commonly suffer from problems such as “ghosting”, where movement of foreground objects within overlapping parts of images results in the foreground object having a ghosted look in the final composite panorama background. Therefore, it is important to extract only the background content of images in order to create a high quality, clean panorama background.
A number of panorama background generation methods have been proposed. For example, the publication entitled “Adaptive Color Background Modeling for Real-Time Segmentation of Video Streams”, by Francois et al., Proceedings of the International Conference on Imaging Science, Science and Technology, 1999, pp. 227-232, proposes an algorithm where the panorama background is modeled from a number of background frames in HSV color-space and subtracted from noisy frames based on the mean value and covariance of each pixel of the static background model. The publication entitled “Foreground Object Detection From Videos Containing Complex Background”, by Li et al, Proceedings of the 11th ACM International Conference on Multimedia, 2003, pp. 2-10, proposes the use of a general Bayesian framework where a number of features are used to model the background for foreground object detection. The main problem that these panorama background generation methods share is that they are primarily designed for object-tracking purposes and are thus, not concerned with the quality of the resultant background content.
A number of slower but more accurate algorithms have been proposed for generating higher quality panorama backgrounds. For example, one such panorama background generation algorithm makes use of an exact pixel selector that compares images pixel-by-pixel to isolate the background from the foreground. Unfortunately, these more accurate panorama background generation methods are inflexible, require that no noise be present in the images of the set (thus making them impractical for real-life situations), and yield poor results when only a relatively small image set is available (which is common in many panorama background generation situations). In addition, these panorama background generation methods are computationally expensive, especially when a large number of images is processed.
A basic assumption with all of the above panorama background generation methods is that the background itself is stationary amongst all images. As a result, these methods are generally intolerant to image misalignments and distortion, making these methods unsuitable for panorama background generation in situations where the background itself is in motion relative to the camera.
A number of deghosting algorithms are available specifically for panorama background generation purposes. The publication entitled “Construction of Panoramic Image Mosaics with Global and Local Alignment”, by Shum et al., International Journal of Computer Vision, Volume 36, pp. 101-130, 2000, proposes a method for eliminating ghosts based on computing optical flow and then performing a multi-way morph. However, this method is effective only for correcting minor motion changes. The publication entitled “Eliminating Ghosting and Exposure Artifacts in Image Mosaics”, by Szeliski et al., IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2001, proposes a technique where a vertex cover algorithm is used to selectively remove all but one instance of each object in the overlapping part, thus eliminating ghosting that results when blending multiple images in overlapping areas. However, there is a problem to this approach, and with deghosting methods in general. Most deghosting algorithms require the selection of a reference image to eliminate the need to blend overlapping parts and are generally concerned with eliminating or reducing ghosting rather than generating a clean panorama background.
As will be appreciated from the above discussion, improvements in the generation of panorama backgrounds from a set of images are desired. It is therefore an object of the present invention to provide a novel method, apparatus and computer-readable medium embodying a computer program for generating a panorama background from a set of images.